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programs, lihe 
,cn program initiation.) A 
program modification 
explaiiiing the development 
procedures are then presented 
flow chart. 



^ccompanyiuy «auuDooj^r uir ^^^w foctses .cn p 
flow chart illustrating the different stages of 

and. termina-tion precedes an introduction 

and use of '^^his manuaXj Guidelines and 
for th^ six major stages identified in^ 
are as follow: operate program (initiate evaluation plan) ; analyze 
internal and external factors to deteriine whether or net the 
is meeting original goals and/or objectives; identify cpticus 
modification and analyze conseguences cf each option; recommen 
program modifications ; obtain approval's for changes; and initi 
program changes^:. Appended are two presentations on occupationa 
program evaluation which .were given at the"^ workshoiJs in which this 
manual. vas developed. (Btt) 




♦ Reproductions supplied by EDRS ate the best that cat te made * 

♦ from the original documen.t. " * 



i 



CALIFORNIA 
GUIDELINES FOR 



MODIFYING A^JD TERMINATING OCCUPATIONAL PROGRAMS 




EVALUATION HANDBOOK . 
PRODUCED UNDER PROVISION OF 
P-L- 90-35, EPDA, PART F 




ROBERT E, HOLCOMB ' 
EDUCATIONAL CONSULTANT, 
PROJECT DIRECTOR 



0. 



ERIC 



WILLIAM MORRIS) 
EVALUATION SPECIALIST 
CHANCELLLOR'S (!)FFICE 
CALIFORNIA COM>iuNITY COLLEGES 



WILLIAM j, CALLAHAN 
CONSULTANT, RESEARCH AND 
PROGRAM DEVELOPMENT 
state" DEPARTMENT OF EDUCATION 



JULY 1978 



C: 



U S. DEPARTMENT OF HEALTH. 
EDUCATION A^ELFABE 
NATIONAL INSTITUTeyOF 
EDUuCATION ^ 

THIS DOCUMENT HAS BEEN REPRO. 
DUCED EXACTLY AS RECEIVED FROM 
THE PERSON'OR.ORGANIZATION ORIGIN- 
ATING IT POINTS OF VIEW OR OPINIONS 
■STATED DO NfoT NECESSARILY REPRE- 
SENT OFFiClAL NATIONAL INSTITUTE OF 
EDUrATlON POSITION OR POLICY 



TABLE 0;F CONTENTS 



9 



WORKSHOP REVIEW • ^ ' ^ r 

• GUIDELINES. CHi^T?;:;;^^^ , , l ; , _ 

CONSORTIUM 'iffilffiERS^^^^^^ . . . i. . / j . . . 
INTRODUCTION . .v. . i. . . . ; 

OPERATE PROGRAM: ^INITIATE EVALUATION PLAN 
INTERNAL EXTESNAi FACTORS . . . . . 

INTERNAL. FACTORS". . . . V \ . 

EXTpJgAL FACTORS . .. . . . . , 

IDMTIFY OPTIONS FOR MODIFICATION ... . . 

RECOMMEND PROGRAM MODIFICATIONS^ . 
OBTAIN APPROVALS AND INITIATE PROGRAM CHANGES 



APPENDIX AND BIBLIOGRAPHY 

EVALUATION OF OCCUPATIONAL PROGRAMS . . . . ... 

DR. PHILIP m SORENSEN/ ; 

PROGRAM EVALUATION . . ' J - - • V 

C. ALLEN PAUL . 

" ■ - , • - • V . ■ , 

OCCUPATrONAL ADVISORY GO^TTEES 

SACRAMENTO CITY COLLEGE , ^ 



ADVISORY COMMITTEES . . . 

^ TYPES AND FUNCTIONS 

BIBLIOGRAPHY . [ . . 



OCeilPAtlONAL GUI DELI NE : FLOWrCHARt FOR 



OPERATE PROGRAM: 
'INITIATE EVALUATION 
FLAN 



Review goals and ' 
* , objectives 

I 

' Review evaluative 
system * " ' 

Review criteria for 

data collection 
, . , (External and 
interip^ 

Beg;Ln infopiation 
. collection 



ANALYZE INTERNAL AND EXTERNAL FACTORS TO 
DETERMINE WHETHER! OR NOT THE PIOGRAM I5 
MEETING ORIGINAL' GOALS AND/OR OBJECTIVES 



INTERNAL ANP EXTERNAL j 



Cdmmunicatidns 
Political aspects 
Budgets * 
Legal aspects 
Transportation 
Goveni£mc^ 
Coop, education 
Work experience 
Funding sources , 
Priorities 

_ V I ^ 



Internal ] 



T 



— 1 "':/ 

External/] , 



; Appropria,t^ness of 

goals "and objectives 

' ■■■■ I'v- 
Enrollment 

■ . " '"\ \ V 

Hecruitment 

' t • ~ 

Inputs * from; Students. 
^Faculty 
Administratiojd 
Trustees 

• I." 

Equipment , Facilities , 
Personnel, Qosts, 
Nature of sti^dents. 
Administrative decision^. 
Support services. Time 
of pjffering, Ctirriculum, 
Methodology 

Placement ^ 

Student p<5.rf drmance 



Input from: / 
' Employers 
Labor / 
Parents' 
Other ^training 
institutions 
Trustees • 
Media 

Advisory committees> 
Licensing agencies^ 
State evaluation 
systems 

Changes^ in: 
Technology 
Sociologicy^l asp^ects 
Economic jtactors 



Effect'* on other programs 



IDENTIFY OPTIONS FOR 
MOMFTCATION AND 
ANALYZE CONSEQUENCES, 
OF EACH OPTION 



Establish rationale 
and strategy for 
change 

Determine 

imp 1 emen t a t ion 
plans 

Establish time 
lines 



RECOMMEND ^ 

PROGRAM 

MODIFICATIONS 



Provide 

justification 
■■^ for ^changes ■ 

I* 

Provide priorities 
and alternative^ * 

> I ■ 

Establish proposed 
evaluation methods 
for proposed 
program 



OBTAIN 

APPROVALS FOR 
CHANGES 



Modify, suSpend* 
-or terminate 
program 



INITIATE 

PROGRAM 

CHANGES 



GommvTn'i^ t a 
/ihahges 



, EPDA CONSOiRTIUM 
GUIDELINES FOR MODIFYING, AND TEl 



lERS 12-5-77 ^ ( 
ElfATINe OCCUPATIONAL PROGRAMS * 



p:^. William Allen, Director . ^ 
..Career & Occupational Education 
^tios Angeles. CSinniunity College District 

2140 West 'Olympic Blvd. / / 

r. Los Angeles, Calif. 90006 ^ ' . 

(213) 380-6000, Ext. £74 

i,pr. William Anderson 
Professional Development Speciali's^ 
Division of -Occupational Education 
California Community* Colleges 

Vl?238 "S" Street / 
Sacramento, Califmmja 95814 
(916) 445-0486/ 7 • ' . ' Syj 

Mr. William J. p«Wahan, Consultant 
.Research and Program De\relopment 
State 'Dept. of Education ' 
721 Gapitol Mall ' , 
Sacramento, California 95814 
.(^^16) 322-2330 



Robei;t E. %lcomb 
Project Director^ , . 

2l40' W. ^ Olympic Blvd. 
Los Angeles, California 90006 
(213) 3^80-6000, Ext. 2fel 
Home: (213) 475-5651 

Dr. Chester Howe.y ^ 

Director Inst. Qiieratio'n 

Simi^ Valley Unified '^School iy.strlct 

875 East Cochran. . 

Simi Valley, California 93065 



[r. Albert L. Metzi^r 
'Regional Occupational Programs Consultant! 
State Department cTf Education 
1025 "P'V Sjtreet * " 

Sacramento California 95814 
(916) 322^2175 



Dr.^ Bill Morris 
Evaluation Specialist v 
California Community Colleges 
123at^"S" Street ; ^ , 
Sactiamento, California 95814 ' 
(916) 4ff.5-0486 / 

Ms. Tressie Outland, Director 

La PUente Valley Regional 

Occupational Programs 

P.O. Box 1217 

La Puente, California 91749 

(213) 333-2201, Ext. 361 

Mr. C. Allen Paul 
Dean of Technical-Vocational Ed. 
Grossmont College r 
8800 Grossmont College Drive 
El Cajon, California 92020 
(714)- 465-1700 . 



(Mr. Merle Runolfson," Director 
' i Vpcational Education 
I Hayward Unified" School District 
1 24411 Ainador Street 
' Hayward, CaMfomia 94540 
,(415) 881-2600 

. Mr. John B. Sharon, Director 
Grossmont Union High School District 
I^. 0. Box 1043 
La Mesa, California 92041 
f714) 465-3131 

Mr. Joe Tijerina - 
Assistant Administrative Coordinator 
Div- of Career & Continuing' Ed. 
ROC/ROP * 
i 450 North Grand , Room 429 
, Los Angeles, California 90012 

Mr. Glenn Van Noy 
Laney College 
900 Falloti Street 
. Oakland, California 94607 
(415) 834-57^40 



' ^ • ■ ■ ^ ^ -'3 ^ , ■■ \ - ■ - 

INTRODUCTION A \ 

' ■ ■ ■ ■■,..■,.1 .. . 

Distribution . • . ^ ^ , ^ 

/. This handlTbok is being sent to secondary schools, ROC/ROP directors 
and 9ccupatibnal administrators throughout the state of California, 

This manual was written as the result of a series of "hieetings held in 
nine California ^cities during the spring of 1978. Invitatiyins.'to 
participate were sent to al]^ secondary school occupational administrators 
Cy the State' Department of Education and the Chancellor's Office of - the' - 
State Community Colleges. / . , ^ 

The primary thrusts of the meetings (further described in the 
section following entitled 'jBrief History") we;:e to review an occupa- • 
tional program planning manual written last year , Guidialines for Occupa- , 
tional Program Planning , and to Improve on the* latter sections of that 
handbook — parficuiarly as they related to oqcupationaj-^ptogram evalua- 
tion. / - ^ / ' 

Additional copies of this manual may be obtained from' Dr. William 

^. ] \ 

Morris, Calitomia Community Colleges, 12G8 "S'\ Street, Sacramento,. 
'Calif. 95814 or from Mr. ^William J. Callahan, sAate-Dept. of Ed2i^ation, 
72L Capit^ol Mall, .^acramentOn Calif. 95814. 

Ryief History ^ ' , V ' 

• « " * ■ ■ ' ■ - ^ ^ ■ 

-7-- This handbook was written after a series of r/gional workshops were 
conducted ^\^throughout the state of California as the result of a project 
funded under provisions of the Educational Professions Development Act, 
Part.F, (P.L. 90-35). ; The office for New Dimeiisions of th^ Los Angeles 
Commimity College District' served as the fisSal ag^nt for the project. 



This prt)ject was a "continuation" of /a voca\:ional education^. 

* * ■ ' • '* 

prpgr.amv whic^ had produced, the previous y^axf a nlanual entitled ^Guideline 

i - • , * • . * • • 

fox Establishing, ^ddlfying and Terminating Occupational Programs > The . 

. . • . «- • . ■ 

primary purpose of the E.P.D.A. grant was' to provide a series of in-service^ 

training workshops to better- acquaint occ^ipatiofT^l administratots 

throughout the state with the original *'Guide'lihes" manual and- to 

iiSaprove that handbook. ^ - ^ 

■•■ ■• : '° . . f . ' ^ • • . 

Many of the memb'ei;3 rfff the statewide -consortium who had -be^n if involved 



in. the ISlb-ll project volunteered to setve^^ again; this consortium met three 

' : • ■ . -^^ V ./ 

times 'during the grapt peri^ to direct the progress of ;the program^ IJhi^ 
consortiuBl members are listed on page 1. ' . * ' ■ ' ^ 

■ ■ . A ■ " " ■ ■■■ - : . 

' It became apparent during the first meeting of ,the consortium and * 
«the early meetings of the workshops that attending members were acquaint-<dir 



ed with, and generally satisfied, with the materials covering the initia- 

. - ■ . ■ ' ■ * ' ■ ■■ » ^ 

tion of occupational programs but were desireotig^pf improving those areas 

identified as modifying and terminating oc^cupational progi^ams. Second^ly, , 

■> . ■ ^ 

it was noted at both the, consortium meetings and at the workshops that 

- ' ■ <■ * f • ■ • \ ~ 

evaluation was the key to program modification and that the mljor time 

■ ■ ■ - ■ ' ^1 • ■ y- 

and effort iu the workshdps should b^ directed toward improving prGi^:ran 
evaluation.' ' ^ 

The' consortium provided the format for the workshops; the consortium 
identified the projected areas for ^he wotkshops (Los Angeles, Riverside, 
San Diego, Anaheim, Santa Barbarajg^Fr^smf,- San Jose,' Hayvard , Sacramento 
and Redding); tHt c9nsortium also provided the format fox the workshop^ 



ase^tation by a consultant on ( 



which were to have: an overview film; a presentation by a consultant on 

■ ■ ■ ■ ■ ' < . ■.■ ' . . ' , ■- 

evaluation; a review of ^ the 9riginal handbook; a presentation of a work- 

■ ■ ■ vC ■ r 



ing model of a pifogram modification chart for tjhe workshops to develop.." 
The consortium als^o develop^ a fln^ revision of the' chart after the* 
wgrkshops were concluded which is included in* th^ -^front of this 



• workbook'. . 'lZ\<i''^ 'X - ; 



* Ten tjprkshops were planned .buf onIy,.ttii;ie were^ held (the'schedulad" 
meeting in -Los" Angeles was ■ literally rdln.ed, put) . " The meetings ran 



. approximately^four houjrs 4nd included a slide, presentation/ an 

expert s presentation on 'program ^valuation and seyeraj. houra of w6rk- 

^ - shop Members V time *~in- derveloping*' the program m*o^ chart;. > ;J\ 

vj' * :Three.^o,f the -workshop evaluation experts were from SRI Intetriati.on'kl 
in.Men-lo Park;^^ ^^s^ Shii'ley ^cGill^cyddy provided the Siirst presentation, 

. ' . ' ' . .y^ - ^ . ' ' . . - . , • ' ^ , 

Dr. Philip Sorensen directed* the others with asri^istance in several 
I meetings irom Dr. Geo^g^ Ebey.' Mri C.. Allen Paul, dean of technical- 
vocational education at >^j^ssmont College, present^' . the^ evaluation ^ 
materials at tW'San Di^go^meefing. >tr., Robert Sayette, previously 
dean of occupational education J^D East Los Angeles ColXege, and!^ Mrs, * 
Hopfe^;,liolcomb, previously the assistant to the Chancellor in the- Log 
Angeles pbflfaunity College District, diredtec^^ the workshop meet^idfes wit li 
Mr. Robert Holcomb.'t^ ' I .. . ' ^ 

A - ' ' ■ ■ ' ' ' ^ ■ ' / • 

Use of Chajt \ - a ( 

- ■ . ' , " \ s - ■ • , • . ' 

The basic "irking" document* of thjLs handbook is- the chart includied 
at the beginning of this Tnanual^ .. As used in last year's Guidelines 

^ ' '^^ ■ '"^ ■ • 

handbook, the chart is to be used prlmarijly as a check list for occuj)a^ 
" tional administrators. Sj!^, items will loom^very significantly for ^ 
. some programs but may not be particulsprly im'J)ortant for others. No ^ 

effort has been made to -'prioritize those items listed below each box 

• ■ is^v 

again such prioritizatiij^^-ptll 'vary considerably from pr(3grWi to program, 
and from district to distinct. * - . ' 



'Of ' 
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. .fJff the chatt is ysed in conjtin^Cion: V7ith> t;h^ ibuxdellnes hydboolc ,of 
.* Jlastjyfear, ydrkshbp members Suggested, that item o!nQ^ on /the* n^w chart, "Operate 



T Pr.agram" phobia be sui5tqr-J.mposed upon t^he previous ^boqlcVs 'chart, items 9 and 

•.10.;. ^ - ■ : . : ■ .■ A-"^-- . r ' . ■ 

? BibMography ' • « / " <^ ' 




The primary emphasis of this handbook is upon program evaluation 

. particularly Jas it relates!" to /^odification 'an^ ^.tera 'As stated 

in the prior year' s^handbook. Very littX^'infq^atiqn has been written 

on pi;ogram -Djodif icat'ibn and termlnatiloi^by California, schooi distriqts., 

^. , Thus the bibliQ^raphy i^luded in" thj sTlanikl identifies primafilj^ ^ 

m&terJLals. jreiated to evaluation. Soire^rapects of program modification 

were^ in|orpora,ted in thcv "original" Guidelrnes (jl977) . 

j V^endum "^^"^^"^ J , ' * a t 

— T~ ' . - : ■ --.x ■ '■ ^ . v.". 9' ' , . ...... 

lsit\Mdie.di in the l^atter' section of : this iianflbook are mate/llals 

■ V ' k-" - ' 'L' -.. v.- •• ■ - ' i:^-'. \/ 

. provided by Dr.porensen, C. Allen Paul, Mr. Eobert Say^^tte and 

Sacramento City College*. Some o^Dr. Sorensen's material was presented 
at the workshops; some was provided' to the project director after the 
workshops were completed. Mr ; . Paul ' s material is briefed^ fj-om his San 

. Diego presentation \^d Mr. Sjayet te* si material ^on advisory committees was 
pr^ented ar^^ost workshop^ The associate i^ean of Occupational Educa- 
tion at Sacramento College provided data»on' collage '.^method of 
working with advisory conimi4rtees • J ■ > - ^ v 



/ . 
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OPERATE PROGRAM: 
INITIATE' EVALUATIOl? 
PLAN ■, 



v 



^ ■'•^As vindicated in the introduction, this handbook-^sumes that an 

^'.•■ i • . -' - ■7 ■. ■ . 

occupational program haa been conceived and approved; at i§i nox^ to 

V •' ■■ • , ''^ ' S » " ■ . ■ * 

be* implemented. V . . #• • * ^ 

• - ( ' - ■ ■ ^ — ■ - ■ ■ 

Review^oals kn<#Obiectives u ' ' 



As . ident'if iedf in . the 1977 Guidelines , a ^comprehensive set -of goals 

J \ ^ ' » 

' i ^' ^ ■ ~ - ■ 'z'' ■ ■ 

and objective^ shoul4^be^ determined as each new program is formulated, 

Warkshop ,^at£todee^ stressed that pirior to program initiatioja, a careful 

revfeir of theseJotigirial gP^als, aiid objectives should be made and should 

be kept in iaind throughout , the implementation and -evaluation period. 

Review Evaluative System * • ' ^ 

r " . ■ - . . ■ ' . ' ' ' ■ ' ; ,/ . - ■ "■, : , v- ■ ,. ' ... 

. . Similarly", workshop participaiits^s tressed the importance of 

reviewing, prior to program implementation, th^ evaluation system 

» . ' ■ ■ ' " . ■ ■ ■ " " . 

established^ when the program wa6 beigg formulated. * 
Review, Criteria for .Data Colfection and Begin Information 

At this point, the administrator should have^a good- concept of 
vdiat Information he is going to-need'fojr measuring yhether the 
program's* goals' and objectives are being achieved. He^hould have a 
^ specific 'plan for gatherin^g the information;^ he should know: Wliy he 
'needs the data, what he needs," Kow he is to collect .tihem, when he is 
going to obtain them. and how he pXans to evaluate them. 




ANALYZE INTERNAL ' AND. EXTERNAL FACTORS TO 
DETERMINE WHETHER. OR NOT THE PROGRAM IS . 
MEETING ORIGINAL GOALS AND /OR OBJECTIVES 



♦ Inte rnal and External ' \ \. 

— — t— : — — ■ ♦ . , . 

As these t are used in this handbook, the internal aspects re~ 

late to those within the institution; others are e'xternal, A great^ 

nuihber of the items considered in program evali^ion are shared ones 

* ■ ' • , - • " 

between the , ins titution and the community of are iteins which should 

' * * • - ■ ' ■ ' .• - • ' ' ' 

be considered as evaluative factors both within the institution ^and 
outside of" it; such items ^are ^classified in the chart a;s "internal and 
external", ' . i ' ' " 

Commun i c a t ion s 

An aspect oiten overlooked in program evaluation, communication. 

between institution arid communitiy as well as the communication within 

■ " " ' ■ p ■ >, ■ - 

the Inst i'tut ion often need careful evaluation, {low effectively are 
program's^ -advertized tp studentalj, faculty, employejs^ and community? , 
'Is there some method of ascertjaii^ing whether the media form being -used 
, is'^isuccess^ful? Are there sufficient provisions made foF feedback from 

-all individuals concerned? Is there a system fcp: analyzing the infor- 

■ ■. ' ■ • ■ / . ^ . . ■ • ' ■ . ■ \ , ' ' - . / 

mation gathered on communication? , . ' / v , 

Political Aspects 

* »'■■,' 
Every workshop gave attention to this ever-present feature, of ' 
* ■ ■ .<>..■. ' . . ■ \ 

prograiji building. Administrators more arid mare appreciate thit-'a » . . 

program to ^e successful must have the support, of students, employers, 

instructors, community and boards of education/ In addition, approval 

is often ^eeded from area, state and the federal' government. Utilizing 

'Sophisticated communication procedures,' the prudent administrator keeps. ^ 
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cohstituerits aware of the program's progress and, in turn, keeps informed 

J , ' ■ ■ „; ' ■ ■ ' . 

afS £o the reactions of t'he various golitical ft)rCes to the program. The 
use of advisory coimnitLtees , surveys, -follow-up 'studies, and broad-based 
constituents involvement in . the program have provea effective techniques 
in providing the information* needed by the -administrator who is' sensitive 
to the political scene. « ^ 

Budgets . , 

^Workshop attendees reported that this item was often the most crudial 
item in program contml. The able administrator must be aware of ^external 
budgetary chag|es which, may effect his program — whether these are local, 
state or federal. But closer to the scene. are budgetary changes within the 
^institution and'his own need ,t;o "control" the various costs items incor- 
porated within the program's bud ge-t. It was emphasized that in his 
original evaluation scheme, the administrator must establish a systematic 
method of mainteaining an up-to-date budgetary picture of program and 
provide for constant evaluation of the various budgetary items. 
Legal Aspects 

Although closely identified with, political aspects, many of the 
occupational: administrators at the workshops- felt that this item deserved 
separate treatment. Here the emphasis is upon the need for the program 
di-f^e^or to be aware of current and pending legislation .as it relates 
to the program. Often quoted as examples of the changing scene in this 
regard were the recent federal laws regarding the handicapped, sex bias 
and ethnic factors as ,they relate to both student registration and staff . 
hiring practices. Other examples of legal factors necessitating evalua- 
tion analysis include VEA regulations, environmentraT control factors, 

13 
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safety and health legislation as they affect occupational programs. 

• V 

Transportation 

Although not as significant an item as legal aspects in many 
institutions, 'many problems relating to student transportation were 
identified at the workshops. The decision as to where to hold certain - 
occupational classes is often detetinined by the accessibilifcy and cost 
of transportation. Problems of clasjs scheduling and the growing 
demand .for busses for other purposes have made many administrators 
more conscious of transportation problems when evaluating their programs, 
Govei^nance 

The growing' centralization of control — whether within the local 
district, by state or federal action — was often a subject of concern 
At the workshops. As with legal and political aspects, the program 
administrator must be conversant- with the changing nature of governance 
if he is to be fully aware of the many factors which may affect his 
program. Again, the importance of keeping local and state personnel 
informed as to a program's status was stressed at several of the 

c 

workshops. 

Cooperative Education and Work Experience 

Not all institutions are involved in these programs, but those 
administrators representing schools that had such, felt that these 
items were significant enough to warrant establishing evaluation check 
points. The importance of providing proper administration, good 
teachers and proper control of these prograii|s was stressed by the admini- 
strators at the workshops. Examples were citeii of cooperative education 
programs which rhad been well administered at the institutions and which 
had good employer supervision that had continued to flourish when 




_ {-11- ■ ^ ,. 

programs lacking these elements had not been siiccessfuli 
Funding Sources • 

Although this item was mentioned when reviewing budget evaluations, 
workshop members felt that occupational administrators should provide 
for an* on-going evaluation of the funding sources involved in any new 
program. A shift of policies on the part of VEA, the state department, 
private sources and/or th^ district can have a profound effect upon the 
institution's programs. The need to maintain an awareness of policy shifts 
by keeping abreast of national', state and local trends via periodicals, 
bulletins and professional associattion meetings was emphasized by workshop 
members- 

P rio^rities \ 

At several meetings the concept bf prioritizing was stressed. Admini- 
strators recognize that obtaining funds for new programs is becoming more 
difficult and they have to weigh the various programs against each other 
to ascertain which is the better. In a like manner, an administrator 
cannot evsluate every a^spect of every program and must identify those 
items for evaluation which appear to be most significant. Local districts 
and institutions, often are forced to make similar prioritization decisions 
as they review budgets, legal and political aspects of various programs. 
Establishing early evaluation procedures for such prioritization can help 
reduce the trauma which often accompanies making decisions in crisis 
situations. ^ 
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Appropriateness of Goals and Objectives 

As admlnlst raters* begin . to -e^faluate programs, one area too 

often neglected, according to workshop participants. Is that of 
' questioning the original goals^and objectives. Since goals are 

normally '^determined prio): to gaining .the knowledge that comes with 

the experience of running a program, the participants at the 
^workshops stressed the'need for recognizing that these goals^^ ' 

themselves may need to be revised and that careful program monitoring 

and analysis may Illustrate the need for such changes. 

Enrollment an,d Recruitment , , : 

Another key factor in program evaluation is that of enrollment 

- ■ /, . ; . . 

analysis. Workshop participants identified the following areas 

needing evaluation when the administer is reviewing program enroll- 

ment: the nature of students, legal requirements, numbers of 

students enrolling, attrition rates, causes of attrition, declin}.ng 

enrollment, effect of supportive service^ upon enrollment, nature 

of recruitment and retention pro'grams. .Workshop attendees often 

stressed the nded for utilizing student surveys to identify 

reasons for program retention, attrition, and recruitment strengths 

or weaknesses. ' 

Input 

/ 

As mentioned above, workshop participants stressed utilization 
.of a wide variety of institutional sources when evaluating programs. 
The use of faculty and student evaluatlori forms was identified by 
m^piy as an effective method of obtaining feedback. Administrators, 
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too, ha^ve to be alert to information regarding t;he prograiri received 
from staff members, "higher**' administrators and from board members . 

• s. ■ ■ 

Additional Evaluation Ar.eas , . ^ 

' \ , ■ , V ■■ ■ ' , ■ , ■ , *■ ^ 

Without going into a great amount of detail, the workshops identi- 

fled the following .^^r-e^s as additional ones .that the occupational 

Iministrator must re^view. as* he l^s evaluating programs: 

' '■■ ■ ■ ^ ■ . ' . • ' ' ' • ■ ■ ■ • , 

Equipment and Facilities - Cost, adequacy, obsolescence, maihte- 

nance, replacement 

Personnel — Competency/;, tenure factors, costs, strengths and 

weakness,' part time or full time 

Costs - Cost effe^ctiveness, cost benefit, current and pr^ojected o ^ 

Nature of Students - Are evaluation procedures established to evaluate 

students' attitudes, skills, sex, ages, ethnic composition? 

How does one^^change program to meet changing nature and needs 

' of students? ^ • 

Administrative Decisions - How is program" affected by administrative 

pressures? Decisions)? 

■■■ \ ■ ■ 

Support Services ~ Evaluation should be made of the effect of support 
services (counseling, student activities, financial aids. 



guidance, placement, work study, etc.) on program. 
Time of Offering - Could program being offered at better time of day? 

Should consideration be given >€o evening or week-end offering? 
Curriculum When reviewing curriculum, the following questions 

sKould be raised: Are courses sequential? Does curriculum 

really fit student needs? How does this pro'gratn dove-tail 

with other institutional programs? ' 
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Methodology- Are the teaching methods being utilized meeting student 
-needs? What improvements can be instituted?^ Are facilities, 
equipment and personnel satisfactory? > 

Placementv- Evaluation of placement should include establishing a 
system to de<:ermine: Hpw §re students being placed? Can 

• > itistitu^^^ do more to ensure placement? / , ► ' 

Student Ferf orifiance - Are/ there methods of -measuring how students 
are learning t SKll Is, knowledge, dtttitudes? Apart from job 
readiness, how have students gained from program? 

Effect on Other Programs - How has^ this j^rogram affected other 

aspects of the institution* - student^ draw; personnel, other, 
programs?, ' , * 



EXTERNAL 

7 -: 



Input from " ' 

At the workshops, the participants stressed the importance of 

maintaining communication with many external source^ as, a means of 

evaluating programs. Those identdfied to obtain reaction^ from 

included the following: ' 

EmpXoyeirs - The use oy sufrveys and petsonal contacts with employers 
,were suggest^ed as way of keeping abreast of. employer needs 

; . , ■ ' ■■ A . 1 

and evaluating employer reaction to progr^s. ' 

> * • ■ " ■ ■ ^ ' ■ ' . 

^abor - Although only a. limited number s.of workshop members\said 

that their schoo^^had cXoser relationship with labor organiza- 
ation, they suggested that more compunicaf ion with labor groups 
might assist in improving program evaluations • 

Parents - Here again, attendees stated . that only limited contact ^ , 

was made with parents, several stated that this was a potential 
area for program assistance and evaluation, . ^ ' 

Other Training Institutions - With the growing public demand to 
reduce .unnecessary duplication in education, educators have 

.''9 , ■ ■ ' ■ 

given greater attention through regional councils and thjpough 
informal discussions to the. activities, desires and evaluation 
of other institutions. Experience of other schools in 
similar programs has proven beneficial to /those instf tut ions ' 
launching new prqgrams and in evaluating tt^eir own. Regional 
consciousness has^ grown, publicT institutions h£^ b^cjiie more 
aware of "non-school" training institutions such as corajnunity 
based orfeanizaj^jions (prlv^e and public agencies) and the re- 
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■ ■ " ■ ■ ' . ■ ■•: , A ' • , 

cognition that these institutions too can assist when one 

is, measuring the strengths and weakness-es oi programs; 

Trust^g^ and Board of Education members vary consi^rably in 

tl^ir participatiqn in program management, but many workshop 

administraters emphasized the importance of both providing' j 

infqrmation to, and being aware of the reactions^f, 

these policy making individuals. It was pointed out th^t; • 

V when advance information, is -provided to board* members, they 

are more likely to be moi;e objective ^n their evaluation-of 

■ ■ J ' ■ ■ ' 

new programs. ^ , ' 

Media - Apart froin utilizing the media for advertizing program^, 
occupational administrators can gain valuable information 
regarding. .communi\:y evaluations of programs when these 
are reviewed by the media. 
Y Advisory CommjJtitees Workshop attendees stated that other than 
' the institution 's .own evaluations, the cpmn/uriity advisory 

\ committees oft^en serve y^s the . best . evaluativ^ tools, A two- 

■ page review of ^advisory committee use-'and functions is 

- included in this workbook's appendix. 
Licensing Agencies - Although serving only a l^-mited number of 
occupations, the reports of licensing agencies can 
serve as a valuable source of program' evaluation. ' 
state Evaluation Sysltem^ - Although scheduled to go through son^e 
revisions in tha near future, the /various state-wide 
systems, COPES, DROVE, SAM, TRACE havL in the past years 

* ■ ■ ■ . • ' \. ■ 

served as significant evaluation systepisV With Ifeavy' > 
emphasis upon institutional self-analysis, these sys-tem^ 
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/ usua^y^ provide for a broad review of institutional pro- ^ 
gra^, but "haV||^ also operated tg isolate and look-in* ' 
fiepth^at'a single one (e. g. police sciehce)-^ 
Changgs - Because of the numerous changes^ that can af^ct an ^ 

. ^ • insfitutiori's program which ^re external to 'ins.t4tutipn ' s / / 

# ' ' ' ■ • ^ ^ ■ ' ' . \^ J 

• control, the wise administrator tries to ]ceep abreastr^of thes 

. changes arid utilizes information gathered whin evaluating 

his programs; these chaitges include the following: 

Technology. - Because of the ^apid technological changes in 

today's world, it is incumbent upon the administrator of 

occupational programs to keep a^ware of the /impact of such 

changes upon his programs. As ^Ix^^^aluate^ a program, he 

must consider how technological changes ma^ affect the 

teaching methodology, the physical plant, the students and 

, teachers, and what the possi'ble effect may be upon enrollment 

and placement. 

Sociological Aspects - With the continuing population shifts 

being so much a part of the California scene, it is impera- 
tive that the state's educators be aware of the demographic 
changes in their areas and how these may impact upon his 
^bgrams. How will a reduced population at a '"lower" 
school change his programs?' Will a major ethnic shift 
alter the methods of teaching? How will the emphasis upon 
providing equaltjLty of'' training Ir^ almost all occupations 
for women, handicapped and older citizens affect his progrm * 
and hik evaluation of it? How- will these sociological 
factors alter his hiring ^actices? Counseling? Placement? 
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■. ^ . -18 '■ 

EconomlG Factors t Economic chan'ges are as significant to the educ&tor 
as the sociological ones. A sudden shift" in job opportunities 
can change a marginal program Ipto a dynami'^c one or, converse- 
. ly^Cj destroy it. In evaluating /any program, the educator has 
to recognize^ bi^th the long and (short term economic %hifts; 
'\- a prime example of the effects of 8ych\shi£ts upon programs 

is exemplified in the" engineering-alllj^ areas which h^ve wax^^ *' 

and wansd iiv conformity with the ever shifting economy, 

* • •-, * ' ■ ■, . ■■ . , .J 

Some workshop attendees remarked that these 'shifts ' 



co<ild lead )to 'quicksand for good*^ prpgipams in perio^' of 
, ecpnomic stress if p3/acement became the only key ^detetmine 
whether a program was retained or dropped. 

Another. economic factor to evaluate is. the nature of 

I ' ' ' 

' changing competition to the program from other institutions, 

. ■ # 

organizations or agencies • 
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IDENTIFYING OPTIONS FOR 
MODIFICATION AND ANALYZE 
CONSEQUENCES OF jSj^CH OPTION 



\ 




Establish Rationale and Strategy for Change 

u ' ■ 

Havirig^ evaluatjed a program, the administrator has to determine 

whether to m^intain ^it .(status quo) to suspend it (keep it on the books 
^for future use) to modify "it , or in extreme cases, to terminate it- 
The administrator 'who wishes to modify a program now haj to utilize his 
evaluation information the basis for establishing a rationale for* 
. change. What specific areas appear to be needing modification? What ^ 
are the possijjle changes that could be niade? . What are the various ways 
of making tHese changes'? 
Determine Implementation Plans 

The administrator, having decided what areas need changing ^d - 
having established various alternative methods of operating the program, 
must now establish a plan for change, Are such changed feasible? 
Suitable? J^cceptable? Some road blocks appear immediately if. personnel 
changes seem imperative ~ can one move to transfer a permanent., - 
tenured instructor who has strong ties with one of the teacher 6rganiza--' 
tions? Are costs prohibitive when newer eq\iipment is needed?' Can bi- 
lingual teachers 'be hired when the ethnic make-up of the students make it 
appear that. this would be wise? . . 

Establish Time Lines ' . * 

Along with thd determining how to plan to imi)lement change, the 
administer must decide what schedule to follow in relation to •'selling 
the, changes" and in implementing it after pbta^ining approvals 
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RECO^IMEND PROGRAJt 
>y)DIFICATIONS 



r 



-Provide Justification for Change 



*Ass,uining that these changes must be approved by another adininistrat;or 

the program adminiistraj;or must provide his reasons for making" them. What 

have been the results 6f .evaluation analysis? What will result from 

these changes suggested?' What are the effects upon personnel, students, c 

' ■ ■ ■ . ■ ■ ■ . ■ ' •. • ^ ' ' ■ 

facilitie^, cost, equipment? Are there income factor^ to consider?- « 

y 1 ' ^ 

Legal or poli^tical consequences? ^ 
Providg PAioijl^ies and Alternatives . 

Th^y^administ^er should ascertain which changes are most signifi- 

Iran t and when each should be implemiented . According to the situation, 

t ■ ^ . ■■ ' : - 

often altepiate change strategies should be provided when costs, personne'l 

or other significant change factors must be considered.. . Many of the 

cjbnsideratiorfs incorporated In the original Guidelines' riattdbook in program 

initiation should be, reviewed if .major changes are envisaged. 

Establish Proposed Evaluation Methods for Proposed Program J 

Included inr. recommended program inqdif ications should be newly 

deter^fei^^d evaluation plans. If extensive changes are proposed, considera 

tion should also be given to reviewing and/or revising the original goals 

and objectives. ' 



^ 

OBTAIN ^ 

, APPROVALS FOR^ 

CHANGES 





^ JNITIATE 


■J , 


^ PROGRAM 




CHANGES 



M6dify,> Suspend or Terminate Progr^ 



As directed by the board, president or principal , the occupational 
administrator should receive approval for ch^ges and then implement 

^ - ' "N _ . ■ .. . ; 

them, .Suspended programs are usually those kept "on ice" fbr a y^ar 
or two when more .favorable conditions may provide, an opportunity to 
reinstate them, ^ ; 



Communicate Changes 

A final area of workshop discussions centered on the need for admini 
stratorg to fully inform students, staff and community of program changes 
There was general agreement that efforts to communicate more extensively 
woi4JLd result in more effective' programs, ^ ' ^ ^. 



Although the above paragraph completes the coverage of the chart 
development provided by 'the administators at the state--wide workshops, 
the following materials describe in greater detail the contibutions of 
the consultants who assisted with the project. This appendix includes 
an evaluation study by Dr, Philip 'H, Sorensen, an outline of Mr, Allen 
Paul's .San Diego workshop presentation, an overview of, advisory committee 
functions as provided by Mr. Robert Sayette and a bibliography of ^ 
materials primarii^f related to program evaluation, . " 



eVAJLOAtlp^ OCCUPATIONAL PROGRAMS 



J)r. Philip H. Sorenseri 



Introduption 

■ . ■ '' ■ 

Planniag, implementation, , and evaiuatipn are recurring functions of 
pr:bgram management . A program is planned as 'a -reasonable- solution a 
prx)biem or need — as a waj^ to overcome a difficulty \or achieve certain 
goalg. The planned program 'is tftBd out or impleihented to see vhetHer 
the problem is solved or the goals are realizjed.'- 



Evaiuatipn is tiie process of assembling evidence abput what i/as done 
and about the consequences of what was , done so . .that sensible judgments^ 
^and decisions can be made about what ta do next. The judgment s^ b'^sed 
xlpoh -the evidence may lead to a decision to try again but in a soniewhatv 
different way. Thus/ unless^ the initial , problem or need, that started 
thie Whole process evaporates in the meantime, the sequence of planning, 
implementation, and evaiuatipn begins gain but with more and better 
information on which to base the new planning. 

^ This sectiM is' about the evaluatiidn of occupational programs. The 
purposes of the'C section are limited to the: f ollawing main points: 

1. To show how evaluation is related: to program planning and 
program implementation. - ^ . T 

^ ■ . : ■ ' ■ \'i ■■■ 

Z. To id entify some desirable qualities to seek in" any program 
evaluatiGn. . . ^ ; V 

. 3. To clarify some r^asphs for evaluation anid to illustrate how 
various reasons lead^' to differeht approahces. ' 

'4. Xo describe various recognized and accepted models or 
approaches t& program evaluation. 

5. To list some references or other sources that provide more 
detail than is appropriate for this handbook. 

Uses pf Evaiu'atidn , ^ 

In the broadest sense, the purpose of evaluation is to helg someone 
^make a depision or choose Bmong. alternatives . In the context of occupa-^*-- 
tional program planning, tne central decisions are about ways to, improve: 
the total 'program and individual elements of it. » 

Sequentially, the evaluation question of first* priority is to decidel 
whether or not specified program purposes are important enough to justif)^ 
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some expenditure of resources. Note that this first question is asked, 
of the purposes to be served by a program, not of a p^gram as it 
currently operates. This allows one to assess the importance of purposes 
•independent of any existing or proposed means for achieving such iJurposes, 

Figure 1 illustrates a sequence of questions that an ocpupational 
program manager might ask with respect, to a specific course or othar 
element ill the oversall program. For illustration/ let that element be . 
denoted as "X." 

A|| Figure 1 suggests, the first question that systematic evaluation 
can help answer is whether or not certain program purposes are wjorth . 
pursuing, , The approach to such a question is not unlike that followed 
in program planning and somehow distinct from evaluation, WitK equal 
justification, one may see needs assessment as a first step in evaluation. 
Functionally, such distinctions are not worth an argument. Essentially ^* 
the same tools and techniques of formulating questions, gathering and ' 
analyzing data, and reporting findings apply to the process. The 
important point illustrated in Figure 1 is that the termination decision 
is based on an assessment of the importance of purpose^ of '*some" program. 

If it is established that the purposes justify some effort, then 
it is appropriate to assess the appropriateness of alternative means to 
those ends. In .Figure 1, a means (program X)' already exists, so f^e 
next question seeks to determine the adequacy of X. If the purposes are 
worthwhile ("yes" to question 1) and the present means- to those purposes 
are satisfactory ("yes? to question 2), then continuation of X without ' 
.change is justified^ * 

Figure 1 has been intentionally simplified to illustrate only 
yes-or-no answer^ to broad questions. JLn. prkctice , answers are not 
likely to be categorical— agreement or disagreement will be expressed v 
In degrees or shades of difference. It '-s a rare program in which some 
modifications will not be considered desirable, either in the specif ica~ 
tion oi purposes and objectives or in- the way that the program operates. " 
Thus, the cycle of planning modifications, trying them out, and evaluat- 
ing them is initiated. In the end, the main application of evaluation is 
in the continuing search for more effective and mbre economical ways to 
achieve worthwhile purposes; ■ 

Evaluation in Relation to Planning and Program Implementation 

4 vPigure 2 illustrates t?wo paths from plan to program to program 
outcomfes. The broken-line path illustrates intentions and goals^while 
the solrd-line path illustrates the observable world. For simplicity, 
the antecedents of "the plan" are not shown — the problem to be solved, 
the needs as-sessment^ that identified discrepeancies between the existing 
and to acceptable state of affairs, and so on. Figure 2 helps illustrate 
several relationships and possible discrepancies that either.may be foci 
of evaluation or factors to be considered in an evaluation. 
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Are the purposes of X 
^sufficiently important 
to warrant expenditure 
of effort and resources? 





Terminate 
X 




Is X satisfactory as it is 
presently defined and operating? 





How can X be modified to 
become satisfactory? 





Continue X 
as before 



Initiate cycle of planning- 
implementation-evaluation 
of alternatives to or, 
within X 



Figure 1 ^ 

SIMPLIFIED sequence' OF QUESTIONS AND DECISIONS ABOUT 
PROGRAM TERMINATION ANi) MODIFICATION 



3 
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PLAN 



INTENDED 
PROGRAM 

9 




ACTUAL 
PROGRAM 



— -0- - 



DESIRED 

purpoMES 



OBTAINED; 
OUTCOMES 



^'igure 2 , , 

BASIC SEQUENCE FROM PLANNING TO 
IMPLEMENTATION TO OUTCOME 
(Numbers Defined ift Accompanying Text) 



between 
hensive 
character 

1. 



Path segment i refers to the. logical consistency' or congruence 
Deuween a stated plan and descriptions of an intended program. A compre 
hensive plan for a program will include at least the following 

rhararfceriStics: 

J. -I 

Specific statements o£ applicable policies, puiiposes, and 
objectives. . - 

Approaches (methods, strategies) to achieve the stated purpdses 
and objectives, described in sufficient detail to guide action. 

Provisions for contingencies, such as an alternative method 
of staffing if ceirtiain key personnel are not available. 

Consideration of environmental factors an^ other situational 
variables that may affect implementation, such as estimates of 
the ^X5b market. 



2. 



3. 



4. 



5. A realistic timetable oi activities and events. 



Specification of individual and organiational 
for Implementation (e.g-, who is to' be respons 
accountable for what actions). 



,al responsibilities 
--ible and held 



ter 
not 



7. Acknowledgment of the relationsiiipg between purposes and . 
resource allocations or, more siniply, a "program budget .'V 

A plan for a program may be evaluated logically against such cri- 
ia ^s the above, it also may be evaluated empirically on many, if 
all* of its elements. For ^example, assumptions should be questioned 
and vertfl^^ when possible, such as assumptions that certain resources 
will become available on schedule or that estimates of the marketplace 
are based on accurate and up-to-date indicators. - 

■ id 



. Path segment 2 in Figure T refers to the plausibility or likelihood 
of achieving desired outcomes given adequate implementation of an intended 
program. The obj'ectives of a' program ~ shown in the figure as ^'desired 
outcomes" should be realistLc, reasonable, and consistent with known 
facts. In a formal sens ^, the desired outcomes are hypotheses about what 
will occur under specif ied circumstances — "If X occurs under <:ondit:ions 
A, ^B, and C, then Y will result." 

An "if~then" firoposition such as the above accomplishes the following 
things: ' . 

1. The intended program — X — is described. 

' ♦ * ' ■ ; ^ . ■ ' ' 

2. * The environmental and situational ^variables'' thought to affect 

the operation of the intended' program — A, B, C, and so on — - 
are defined. * ' 

3. The 'desired outcomes or objectives — ^ Y — are specified. 

Specifications such as the above set the stage for systematic evalua- 
tion by indicatinjg the kinds of questions that should guide an evaluation. 
Was the program implemented as intended? Did external variables exert 
expected influences? To what degree were desired outcomes realized? 

Cqjisideration of path segments 3 and 4 in Figure 2 will help clarify 
the relationship between the propositions about the planned program and 
what actually occurs. Path segment =3 refer^^to the relationship between 
the program plan and the program that ,^iiluai$y 4s Implemented. The 
empirical tesft t>£ the adequacy of ^ the program plan for implementation 
lies in the discrepancies between the intended program and that which 
is implemented (see 5 in the figure). In what ways did the actual program 
depart from what was plann^id or intended? Was a discrepancy between the 
intended and implemented program the result of something that pight have 
been controlled that is, anticipated and^provided for through an 
alternative in the plan, such as a contingency plan for staffing? Or was 
a discrepancy the result of something that was beyond direct control, such 
^as an unexpected turn in economic conditions that affected revenue? In 
any case, what lessons for planning can be drawn from the fact that the 
actual program differed from what was intended? ' 

The importance of identifying and analyzing discrepancies between 
intended and actual programs' highlights a commoii shortcoming in many 
/program evaluations. " All too often, the program that is to be evaluated 
is defined by little moire than its label or name rather than by the 
process, activities, and conditions that give it operational meaning. 
It follows inevitably that discrepancies between intended and "actuspl 
program implementation cannot be detected, described, and assessed -for 
their importance unless (1) the intended program is^ described and 
(2) the actual program is observed and described in similar terms. 

Evaluation research over the past few years suggests strongly that 
certain differences between plans and intentions are unusually critical 
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for their effect on desired outcomes. One such difference is the dis- 
tinction betv/een "planned time on a^task" and "engaged . time on a task.^' 
For example, an instructional program may call for 30 minutes of drill" 
ox practice on a specified skill during each instructional day. A - 
formal schedule and lesson plans may^exist to.help assure that this 
occurs.. Observatibn of classroom activities frequently shows, however, , 
thmt the actual amount of time devoted to the activity is legs than 
planned. The reason may be as simple as the time of day that the activity 
is scheduled, such as preceding or following a break or recess which 
involves some "get ready" or "settle down" activities that consume time * 
but are irrelevant to the instruction. Furthermore, pupils vary In * 
their attending behavior or "engagement in the -fask" within a sinj^e ^ 
classroom. Such variatibn^, both within a classroom and across class- 
rooms, cahnot^ be detected and recorded unless .they are observed syste- 
matically. If one seeks to understand the relationship between instruc- - 
tional processes and their outcomes, then the instructional processes must 
be measured as accurately as ^the outcomes . * • 

• I!ath segment 4 irf Figure 2 illustrates the latter point. Wherther 
or not the actual program corresponds tQ the intended one, it is piily 
the actual program and the obtained .outcomes that can be^,observed, * 
measured, and analyzed to provide a basis for understanding why "X leads 
Cor does not lead) to Y under conditions A, B, arid C." The relationships 
implied by path segmeht 4 are the h|Bart of evaluation reserach — the ^ 
search for explanation in terms of cause and effect. 

Finally, the differences between desired outcomes and obtained 
outcomes Csee 6 in Figure 2) identify the crucial contrast for estimating 
the effects of a planned program. Estimating effects, however, may 
require considerably more than a simple comparison. Suppose that desired 
outcomes or objectives were defined in terms cjf changing an existing state 
of affairs into a mor^ desirable one, such as might bje derived from a needs 
assessment. For example, imagine that a vocational prog^pam manager 
receives reports from employers that many graduates of a vocational pre- 
paration program are not able to do certain things that aii employer 
reasona^bly can eatpect a new hire to be prepared to do following completion 
of the vocatlorTal program. These reports disturb the program manner who 
seeks evidence to support or refute the contention. / 

Some hasty fact-gathering confirms that -the employers probably are 
correct, so the program manager and instructional staff plan a modifi-r 
cation of the instructional program to rectify the oversight. A test of 
the skill is developed and administered to a sample of students currently 
in the program to estimate existing skill levels. A criterion of pro- 
ficiency at the end of training is defined by instructional staff with 
the help of the advisory committee^ and a panel of employers. Instructors 
are asked to implement the plan^^ modify instructional,' and trainees are 
given the performance test following completion of the program. What 
are the possible outcomes and what conclusions might be drawn from the 
result^? . • 



For simplicity^ possible outcomes may be reduced to four extremens 
as shown in Table 1. 
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^ - Table 1 

* • V - • . ... 

INf^RENCES FROM RESULTS OF PROGRAM MODIFICATION AND 
'MEASURES OF SKILL ACHIEVEMENT 



Implementation of 
Program Modification 

Modification Implepiented 
as planned. 

Modification not - 
liuplemented as 
planned, ^ 



, /observed Consequence on Skill Measure 
Cflteripn Level Criterion Level 



Achieved 



Progr'asn appears 
tOi be effective. 

Progtam probably 
is. not a critical 
cause of skill 
achievement.- 



Not Achieved 



Program, does no*t 
appeltr to be effective. 



Program effectiveness 
cannot be assessed 
because the plan was 
not implemented. 



Note that only one of the four concfusions in Table 1 can be stated 
with confidence; that conclusion: is the one which asserts that the pro'-- 
gram modification has not been tested due to failure to implement it as 
planned.' The remaining 'three: conclusions are tentative at best, given 
the conditions of the evaluation! ' ' 

, To assert cpnclusions With, greater ^conf idence or greater generality, 
future evaluations should consider such issues as the following: 
" ' . .' ' ' ' ' ' . ■ ' ■ . 

1. Are there other progiram modifications- that are more effective c 
or less costly or both? , ^ 

■ ■ ■ ' i 



How^mucih of the improvement can be jattributed to influences 
other than the program modification? For example, were the 
asual" or not typical in some key respect? 



trainees "unusui 



3. How valid is the measure of skill achievement? For example, 
: how well does performance on the test predict performance on 
thefjob? For that matter, how critical is the skill- itself 
to overall performance on the job? After all,, the program ^as 
riodified on the strength of employers' opinions, not a- rigorous 
analysis of job requirements. 

^ Several additional questions could be listed, but the above are 
sufficient to make the following points: ^ ^ 



1. 



Measures of program Implementation are essential if one seeks 
meaningful conclusions about cause-and-ef f ect relationships 
between i)articipation in a program and accomplishments follow- 
ing participation in that program. ' ' ^. 

Several reasonable alternative program^ —at least more than 
one -J- must be examined- before any conclusions can be drawn 
about relative effectiveness or relative costs or both. 
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, 3. Conclusions about effects are very senm.tive to qualities of^ 
. the 'measures of effects. When the index of ^training program 

effectiveness is performance in a next higher level of training 
or on the job, tY\^ quality of measurement .validity is particu- 
larly crucial. Furthermore, when effectiveness is assesajpd 
against a standard gr criterion of competence that is baied on \ 
•Judgment without substantiating empirical evidence, inferences 
about effect ivess are bound ditiBctly to the quality of, the 
Judgments that defined the criterion level. 

' 4. The desired outcome of participation in a program rarely is 

singular. Usually one is interested in more than knowlege or 
skill, although these ma^Nie the central objectives. . A progr^ 
may be Judged* against several criteria, and it is not -Unusual to 
find that Judgments about the worth of a program will- Vary with 
the criterion, invoked. Cost, of course, is an obvious criterion 
to Include with a measure of effectiveness. So, tod,^ are measures 
of opinion and preference, including the attitudes of instruc- 
tional staff. Most of these criteria call for measures .other 
than tests. V > 

5. /Evaluation designs and methods of analysis are available that 

will permit more worthwhile conclusions to tfe drawhv For , example, 
alternate miethod comparison designs with either experimental or 
statistica'l contjrol of trainee assignment to methqd rwill . increase 
the generality of conclusions. As another examplev- teplication 
(i.^e., repeating the trial with sucoessive groups of trainees) 
will add strength to conclusions. As a further example, multi- 
variate correlational analyses that take accpunt •;,df trainee 
charateristics, instructor charac4:eristics, specific processes, 
in the instructional method, cost factors^ and 3Q on, will 
extend the range of possible conclusions. H.. 

■ ' . - '^-V ' 

Desirable Qualities in Any Evaluation 

- < . '•- * ^' • 

The foregoing discussion of the relationship of evaluat*Lon to pro- 
gram planning and program implementation has helped illustrate several 
considerations that should be made in planning and iari^ying -out an evalu- 
ation. In the following paragraphs, these and other considerations ar|e 
stated or restated as reminders about qualities to seek in any evaluation. 

• ^Evaluation involves the application of human ^judgment to 
evidence. Because this is so, individual preferences and 
predispositions will influence wayj^ in which evaluation 
questions are stated, selection of evipence, choices among 
methods of data collection and analyses, and so on. A "good" 
evaluation will be explicit in statfhg assumptions and^ 
expressing the rationale for choices among /approaches or , 
kinds, of data. A good evaluation also will be designed to . 
protect against erroneous or unwarranted inferences by ^ 
remembering that a hypothesis can be tested only under con- 
ditions where it can be either rej ec ted /dr^ accepted. 
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Evaiiiations have no value if they stre not available to those <f* 
who must make decisions when the -time for tho^ decisions 
comes due. Pvalua^jion Reports must be timely. This often 
means that approximat'ions must sjtahd for mm:e precis^ estimates 
that might be possible" with more tlme^yr^adikttional ' resourced \ - 
Put anotherway, good evaluations areVdesigneoNwith the 
timing needs of decision-makers in miW. If CMs means 
that compromises in' methods are made tox^ajiriffy a decision 
timetable*, then a good evaluation will be explicit in de- 
scribing the delimitations and restrictions that apply to: 
Inferences that can be drawn from the analyses. 

Evaluations shouM r^ognlze and acknowlege multiple causality . 
of events In general and human behavior in particular. Short- 
term conse^uenc^^may exert a ripple effect on later consequences 
One program may influence other programs in unintended ways. 
Events and conditions external to a program '• — social, and 
political developments, activity in the marketplace — • will 
Influence the program environment, often in i^iexpec ted ways. 
A "good*' evaluation scarpely can be other than multivariate 
in forms, particularly when it is realized that experimental 
controls are virtually impossible to establish and maintain. 
It is just such inability to control conditions in a "natural 
experiment" and the inevitability of multiple causality that 
makes it essential that the programs^ under atudy be observed 
and measured periodically and frequently.. One liay not be 
able to control the treatment, but one can describe it in 
process terms. 

Evaluation data should be based on measures obtained under 
conditions that reflect attention to criteria of me,asurement 
quality. Many of these criteria are too complex to^^jdiscuss 
in this brief section, but mention of them can at least 
serve to alert one to the issues. Key criteria include the 
following: — 

-T Objectivity, or the ability to share data and test for 
common interpretation by different observers. 

~ Reliabiltty, or the qualities of consistency and stability 
in measures. * < 

— Validity, which may refer to several attributes. In 
evaluation of lines tructlonal programs," the following two 
are most important: (1) the content validity of the 
measure (e.g., does a test used reflect a fair sample of 
the content of instruction) and (2) the predictive validity 
/of the measure .(e.g., the relationship between the 
measure, such as a performance test of skill following - 
training, and a ^ubsequerit measure of performance in the 
s^^ttlng where'^^the skill is applied, such as error rates, 
on the job) • 
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— Practicality, or the feasibility of applying^^ the measure * 

or obtaining the data in the intended way. 

■ ■' ■ . • . . . ■ /• ' ' 

. Evaluations must reflect high standards of professionalism: 
(1) the integrity of the evaluation staff, (2) protection 
against systemhtic bias in the formulation of questions, 
aollection of data, and analysis and interpretation of. findings, 
(3) strict adherence confiderxtiality of sensitive or 
personal Information, 

' - * ■ ' . • ' • • ■ ' ■ ■ ■ 

Contrasting Orientations Toward gvaluat ion * 

' ■ ■ . V 

The manner in which program evaluations a?:e approached and carried 
out will be affected by a host of factors, some of which are determined 
by external circumstances and others of which simply reflect the capa- 
bilities and styles of those undertaking the evaluation. Following is 
a brief discussion of jJBte of^^the issues that an evaluator should con- 
sider since each will l^^p a bearing on the manner In which the work is 
undertaken. The issueJiPre expressed below as though they were choices 
between one orientation or a competing one; in practice, the choices 

may not be as. extreme as may be implied, 

. ■ , I ' ■ . ■ • ■ •- ■ - ■ 

Formative or Summative Evaluation 

< . . ■ * - - 

Formative evaluation concentrates primarily on the development or 
alteration of a program, course, or unit while it is still fluid. 
Empirical research methods may be applied to such activities as (1) successive 
tryout and refinement of materials, (2) developing descriptions of target 
groups (e,g. ,> capabilities^ attitudes , interests) to aid in planning and 
materials development, (3) definition of goals and objectives to assure 
clarity, reasonableness (i,e., capable of being attained), and appropriate 
specificity, and (4) implementation procedures and variations in methods. 
The formative evaluator^ sh'ou^ be part of the program development team 
from the outset. As such, the formative evaluator usually shares (or 
comes to share) the developer's enthusiasm for developing something that 
works well. The formative evaluator 's special contribution \jpomes primarily 
from abilitl^ to apply empirical research techniques, in continuous, fast- 
feedback, assessment of alternatives before' the program (course, unit) 
is frozen and put into general use, 

Summative evaluation is directed toward assessing the overall pro- 
gram after -it is in operation,. Summative evaluation* has essentially the 
same meaning as "impact assessment,!' and will pay attention to both 
intended and unintended outcomes of the program, ,If nothing else, 
summative evaluation is concerned with consequences of programs, but 
this does not mean that sui^ative evaluation cannot also involve analyses 
of processes. 

One implication of extraordinary Importance .follows from the dis- 
tinction between for^pative and summative evaluation. It is very difficult 
for the same. person qr team to serve both the functions of formative 
and summative evaluation. The formative evaluator has been an active 
participant in the development effort. The summative evaluator should 



be Independent of the^ program development so that the summitlve evaluation 
can be objective and accepted by others as such. . This, is important, 
because the audiences for summative evaluations usually are those who 
set policy and determine resource allocations . These distinctions 
suggest that summative evaluation should be conducted by a third ^arty , 
who has no stake ^4n th'e decisions that follow the Valuation, 

Comparisoniior Absolute Standards 

Comp^ative evaluations concern relative differences, such as V 
differences* between similar groups or diffeirences over time for a single 
group. Absolutist evaltiations refer n> assessing results against 
standards related to program objectives'. Comparing a group's performance 
against norms on a staridaydized test of . achievement * illustrates one 
common comparative evaluation. By contrast, minimum competency" testing 
(in which a minimum acceptable level of performance Is defined in advance) 
illustrates a kind of absolutist evaluation. In the testing field, the 
distinction between norm-r^erenced tests ^d criterion-referenced tests 
is analogous to comparative vs. absolutist evaluation. 

Internal or External Evaluation , ' ' ^ 

Internal evaluation refers to that conducted by persons from the 
program staff . As noted undef\ discussion of formative and summative 
evaluation, formative evaluation by it^ nature is internal even though 
someone^ "from the outside" is retained to work with the developers and 
provide formative evaluation services. 

External evaluation refers to that conducted by someone independent 
fromi the program staff. Credible summatiye evaluation usually is extertial 
evaluation. - 

■ > - ' ■ . i 

Evaluation a nd Evaluative Research 

V, ' ■* ... 

The distinction between evaluation and evaluative research (or 
perhaps more appropriately, between evaluation and scietific inquity) 
involves issues too complex to develop in thl^ 'handbook. -Roughly, however, 
such pairs of terms or phrases as the! following will connote, if not 
satisfactorily clarify, kinds of distinctions intended: , question Vs. 
hypothesis, descriptive vs. explanatory, impact vs. causal, empirical 
generalization vs. theory, assertion vs. proof, applied vs. basic. 

The above terms imply differences in purpose and approach that \ 
exaggerate some of the distinctions intended. As noted several points^ 
earlier in this section, evaluation is inquiry applied to making choices ' 
or decisions. For example, an evaluation may seek to reduce uncertainty 
for a manger who must make practical decisions about policies- and the 
allocation of resources. Evaluative reseai:ch ^- and to an even greater 
extent, scientific inquiry — is lesar concerned with imminent decisions 
or choices and more concerned with constructing a network of propositions^ 
and theories that "explain" gerieralizable phenomena. 



.y In a crude sense, evaluation must try to an'swer such questions as 
"does • it work" or "how well does it work">.or "how much does It. cost to 
operate." Evaluative research tries tp answer a few more Questions, ♦> 
such as "why does it work that way" or "for ;^hom does it work best und'^^t^ . 
certain conditions" oi "does it hav6 to cost that much." _ . *</ 

Evaluation is orie'pfbc'edufie.. for formulating and testing hypotheses, 
but not necessarily 'hypptheses 'of the same form' and generalizability as 
derived in formal acientlfic inquiry. Evaluation of educational programs 
uses most of the methadological and ^alytic tools of the social and 
behavioral sciences, arid shoti/ld use them with n less rigor and tough- 
mindedness. Evaluation results caiv cpntribute to scientific theary— 
building, but that is not* an essential quality of evaluation. 

Experiments, Quasi-Experiments, and Natural Variation 

The^^ comments might have been headed , "consensus and controversy 
iil^ the design of evaluations." On two fundamental points, there is 
general agreement regarding evaluation design that is, the conditions.- 
and schedule under which data are. collected or measures are taken: 

1. The design should be such that findings are interpret able. 

2. The design should be such that findings are general|izable. 

Interpretability refers primarily to the ability to make appropriate 
"attributions of ca^se." Were the resultsj^^ due to the program, or could 
the results be due to 6ther factors that the design did^ not control for 
or that the analyses could not offset? Examples, of problems of attri- 
bution of cause were mentioned earlier in the disxzus^ion of evaluation 
in relation to program planning arid implementation. 

Generalizability refers to the ability to extrapolate findings from 
a particular evaluation to other situations. (This quality sometimes is 
referred to. as the external validity ^of the design.) Problems , of general i 
zability are entangled with issues of sampling in the broadest 

sense the representativeness of the setting or situation, and the 

replicability of the program. .. ^ ^ 

, ■ . ^ ■ ■ . 

Points at issue regarding appropriate and acceptable designs are ^ 
not trivial. While authorities tend to agree without controversy regard- 
ing the inappropriateness of some designs ,(e. g. , the so-called "ex post 
facto experiment"), i^trpng*po,siti6ns have been declared regarding true 
"experiments" and the place of correlational studies in program evaluation 
Campbell and Stanley (1963) are committed to the true experiment , express 
qualified support for other designs, and sugg^est" rigorous strictures 
regarding conditions that make\ correlational studies appropriate. 

- ^- V ■ ■ 

By contrast, Cooley and lf!ohneH^i976) have made , a persuasive cas^ 
for multivariate correlational studies: "The multivariate correlational 
studies proposed. ..may not suppott the direct, causal inferences that 
some efducators desire, but neither do they make impossible demands for 



rigorously 'axperimental data bases ... Educational innovations have to 
be evaluH^id in natural settings ... Replication should be the rule. 

• .The reliabil/itj^ of important contrasts between worthy, competitive 

ins truction^I^^^madels should be demonstrated by replication of them, over 
and over again if possible. The role of statistical procedures in 
establishing-' evaluation contrasts is heuris^tic more than inferential v" 

i ' ' ■ ' (■ • f 

>-••... . . ' ' ^ . ■ . . .. 

• : The underlying preference reflected in comments throughout this 
section of ||ie hai\^dbook is toward the. Cooley and Lohnes pofiition. 

Genera;l Models of Evaluation 

After giving due credit to others who have undertaHen a similar 
exercise, Ernest House of the University of lil,4noi$, has developed a 
taxonomy of major evaluation molfels (Hciuse, ^1978) . ^Ibroduction of 
House's taxonomy' in, this handbook is a convenient wa^f illustrating 
the range of legitimate approaches to program evaluatioh in education. . 
Since the taxonomy is organized; to display differences according^ to such 
dlinenaions ;>s major audiences, preferred outcomes, and typical Questions, 
the taxonomy /4l so may serve as a prelim'inary catalog of options from 
which one might chdose when considering alternative evaluation approaches 
It should go without saying that the taxonomy can be eJip and ed to admit 
new models. And for someone who treats the taxonomy as a catalog of 
options, it should be evident that eclectic combinations are nor , 
prohibited. , , 



; House's taxonomy which follows is as It appeared in the Educational 
Researcher, Vol. 7, No. 3, March 1978. Even though the table .format is 
highly telegraphic in its descriptions of ;the major .models , effort 
: to expan^ upon that , abbreviated jiescriptio'n in this handbook would 
be scarecely less concise. For dreaders who seek more information, the 
bibliography at the pages^ 43 and 44. *^ 
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Dr.' Philip H.^orensen is a seniojf'psyc'hologlst In educational research 
^t SRI International , 333 Rovenswood Avenue, Menlo Park, Galif . 94025^. 
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FIGURE 1 : A TAXONOMY OF MAJOR ViMiiVAVOH MODELS 
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' , PROGRAM EVALUATION - '0 

C. Allen Paul, Deari',. Technical-Vocational 
Education^ Grossmont College ; J 



Directpr's Note: ^The folloirtpg is a brief^outline of the material 

presented to the ' San D^ego X^idelln^sf^ on March 3, 1978 

by Dean Paul on 'the siitject /af .occiipat program evaluation. 

, -.^ ^ ■ • , . • • ■ ■• , 

The riaasons for occupational j>rograiii evaluation can be summarized 
by utilising the cjuestions to be answered when seeking to write a 
comprehensive news story: Why? What? How? When? Who? 

^' ■ ■ . . . ■ ' , \ 

, -The why of occupational program evaluation can be sunhnarized by 
reviewing the following: accreditation, fiscal .audits, VEA requlre- 
merit of program's strengths and weakness, cost ai^aly^is,' tod ..d^ for 
grants. | - ' _ ' I ' . 

■ * l^he what areas iricorpprate the various aspects of the program to -' 
. be ev^luat^d and, among others, include: curriculum, relevancy, metihoH 
dology, instructors, facilities; rifekources, enrollment, placement^ client, 
satisfaction, employer satisfaction and- timing. 

\ The how factors in evaluation incorporate a wide variety of 
techniques-.** Familiar to most' octSiipational educators in California are 
the more formal systems (COPES , DROVE , TRACE, and SAM) , but various types- 
of community, district arid institutional surveys have been effectively 
utilized at Grossmont. Student follow up by the college utilizing a 
•mail system (copy appended) has been used for many years. In addition 
cost analysis of such items, as enrollments and attrition also are used. 
Various methods of instructor evaluation (by students, self or by pedrs)-/? 
have been effective. * : ^ • * 

■ ' ■ , • r . t ^ v;- .; ,;V ^ • , :,;vr..:';- , 

The- question as to when to evaluation is a constant one^ Most 
educators would opt for evaluations on a cpjntinuum ~ have evaluations 
as a^method of locating potentfial problem^, -rather than evaluating after 
trouble as surfaced. Evaluation is done w:hen yoU cati afford it (you 
often c^/t afford not to evaluate) . Evaluation is a tool for improve- 
< m6nt and thus should be built into every prograta as a routine process of 
..action.' ^ .., . 

/ Who are to be evaluated?/ Again the quest ioli ijidy llave a wide yariety 
of answers. According tp the program priorities, certain elements will-' 
get^ci^oser scrutiny than others, bcit the following are listed possible 
areas for evaluation: (students^ employers. Instructors, administrators, 
^ supportive services, Icouusellng, place^ient, etc.) By thei stoe "tok^^^ 
these just-listed elements may also serve' to /do rhe evaluation of ttikir 
.own parttcipation in the program or sblhe aspects of the program itself. 
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Dear Grossmont College Attendee or Graduate: 

The Grossmont Community College Program is vitally dependent on feedback from you and others who have had educational experiences 
here. Your response by simply filling in this brief questionnaire when added to all the others constitutes vital information pertinent to 
* course ^and program development and change. We need to know from you how valid our findings and assumptions have been in es- 
tablishing the present curriculum. 

In the absence of any other means to secure this kind of feedback, I am taking this measure, and ask you to please respond Your at- 
tention and thoughtful cooperation to this matter will be greatly appreciated, and serves those who will foltow. 

f, Sincerely, ^ ^ 

^^^^^^^ 

O SOCIAL SECURITY NO I I I I I I I 1 I I I I C. Allen Paul, Dean , 

_ YES NO — — — ' — ' — ' — ^ — ^ — ^— ^ ' Technical-Vocational Education 

© □ CZ] ARE YOU PRESENTLY ENROLLED AT A COLLEGE OR UNIVERSITY? IF YES. CHECK (,) COLLEGE(S) BELOW. > 

. ^ 3752490 Grossmont College 1, 3756764 San Diego Evening College 3780582 California Western University • 

^ 3767209 San Diego State University 3756939 San D^lego Mesa College 3781473 Point Loma College 

'""^ 3778370 Universityof Calif.. San Diego __ 3753001 San Diego Miramar College 3780582 United States Intemat'l University 

3756632 ' San Diego City College 3758075, Southwestern College . 3788486 University of San Diego 



MAJOR ^ UNITS COMPLETED SINCE GROSSMONT 

© WHAT IS YOUR PRESENT EMPLOYMENT STATUS? ) CHECK ONE 

1 ' Employed in occupaUon trained;'educated; Namely 

2. Employed in related occupation; Job Title 

' 3 Employed out of field; Area " 



2222222 Other 



^4. Employed in an apprenticeship program; Namely _ 
^5 Employed and not satisfied; Reason 



© WHAT.>SINGLE STATEMENT BEST DESCRIBES YoJr PRESENT JOB? {, ) CHECK ONE 



_1 . Employed parl-timp, less than 30 hours. 

_2. Empbyed full-time. 30 hours Or more. 

_3, Unempbyed. Seeking Employment. 

^4. Unemployed, Not seeking work; why 

_5, Empbyed in Military. 



CT\ YES NO 

^ .[ZD dl WOULD YOU LIKE GROSSMONT COLLEGE S PLACEMENT OFFICE TO HELP YOU IN SECURING EMPLOYMENT? 

® DO YOU I ), CHECK ONE 

1, Live With your parents? , 

2, Rent? 

,3. Own your home? 



^ Yes No 

CZ) I f r~~1 Received Cerliftcate from Grossmont; When . . •_ 

Yes No 

C£) I I ^'Received A S Degree from Grossmont; When 

Yes No 

® L I Received A.A Degree from GrossrrxDnt; When [ ^ 

^ Y^ No 

(io) I 1 r~~l I" Armed Forces while attending Grossmont, 
^ Yes No 

[ \ Presently receiving public assistance or plan to apply. 
Yes No 

® r_ 1 Did Grossmont College curriculum meet your needs? 

Yes No 

@ [ I Did you acquire a marKotabto skill at Grossmont? 

@ Area of study in Vocational Education 

® Area of study other than Vocational , 

® NUMBER OF DEPENDENTS 

© PRESENT EMPLOYE n 

COMMENTS. 

ERIC 
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SACRAMENTO CITY COLLEGE 

Occupational ADVISORY committees 

Formation . Each occupational education program shall have an advis6ry committee- 
Programs offered at more than one Los Rios College may operate with a joint advisory 
committee. 

Membership . Advisory committee members will be appointed by the college president 
under authority of the Board of Trustees after consultation with the occupational 
administrator and program faculty. Membership will normally be for a term of 1-3 
years. Members will be selected based upon consideration of: (1) the authority of 
the individual to (represent an occupational "^roup , (2) the active and current involve- 
ment of the individual in the occupational field, (3) interest, enthusiasm and time 
commitment, (4) representation of community and student population. Department faculty 
attend and participate in advisory ^nimit tees as program resource persons. 

Meetings . Occupational advisory committees will normally meet each semester. Special 
meetings'- may be called according to need. The occupational area deans will be respon- 
' sible for develppment of an annual schedule of advisory committee meetings and for the 
review of meeting agendas and minutes prior to distribution. Meeting notices, agendas, 
and Support materials will be mailed two weeks in advance of the meeting with minutes 
being mailed within two weeks after the meeting. Meetings will be chaired by the 
instructional subject area chair or designee. Minutes will be taken by a subject area 
chair designee • 

Role and function . Occupational advisory committees advise school administrators and 
instructional staff on matters impacting educational preparation for employment. Com- 
mittee functions include: 

1. Providing information on specif ic skills needed for successful employment 
within the occupational field. 

2. Alerting the college to available educational field experiences and personnel 
qualified as teachers, substitutes, guest lecturers, speakers, and resident 
experts. ' 

3. Providing recommendations on the instructional program. 

4. Assis ng in improving public relations and communication between the 
college and the community » ■ 

5. Assisting in recruiting students and in providing internships, work experiV^ 
ences and graduate placements. ^ ^ 

6. Providing review and recommendation of curriculum changes for consideration 
by the' campus curriculum committee. 

7. Reviewing legislation and administrative issues impacting the educational 
program and/or the occupational field. 

Parliament ar^^yrocedures . Advisory committees are designed to operate on a general y 
consensus basis» In rare cases formal votes may be recorded. Only^ bonaf ide com- ^^"'^^ 
mittee members may initiate motions and vote. All actions carry the status of 
recommendations for administrative and di3t;rict governing board consideration. 

Evaluation . Occupational area deans will evaluate advisory committee meetings on 
the basis of appropriateness of agenda, general conduct and participation of the 
meeting and principal outcomes. The goal of evaluation is improved instructional 
program performance. 

Associate Dean, Occupational Education 
^>^?/8/77 

ERIC 
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OCCUPATIONAL PROGRAM GUIDELINES 

2140 West Olympic Boulevard, Suite 531, Los Angeles, California 90006 . (213) 380-6000 Ext. 261 

Occupational Advisory Committees 
February 1978 

Types : . 

1. Community Advisory Commi ttees-- generajj ^f^ 

2. Occupational, or trade-tech. Advisory Commit tees-- s£ecialjsts_ 

Purpose: 

to insure the appropriateness of existing and 
Proposed vocational programs or courses. 

Duties and responsibilities: . . 

To advise and assist in the development of -new programs. 

To review existing programs and courses. 

To review content of existing programs and courses. 

To supply technical information. r 

To suggest modifications: 

Changes ip course content ^ ^ . 

Deletion of courses or programs 

Updating of equipment and/or tools. 
. Standards of performaiica (level of skiTls to be developed) 
Selection criteria of students. 
Employment trends immediate and future. 

How many trainees can be absorbed. 
Placement of students in full time and part time positions. 
Assisting i'^ instructor recruitment. 
Development of apprenticeship training (if appropriate). 
Entry level wages and salaries--maximurn obtainable. 
Determining value of program for employees presently employed in the 

occupation. 

Evaluation of texts, manuals, brochures and other instructional 
material. 

Types of equipment, tools and supplies needed. 
Assist with acquiring donated equipment and supplies- 
Information relative to the adequacy of existing facil i tiesor. 
Suggestions for modification of existing facilities. « 
Assisting in the development of new facil i ties. i 

Limitations of Advisory Committees: 

Limited to advising, assisting and suggesting. 

Program operation is the sole responsibil i ty of the educational 
institution. 

Composition of Advisory Committees: 

Representatives of employers; .employees, unions and public 
agencies and. students. 

The Committee should include individuals who can supply technical 
and personnel information. 
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Frequency of Meetings: " 

This will vary with\he task confronting the Advisory Committee. 
New program cfevelopment may require meetings as frequent as once a 
month until the program becomes operational. 

Program moajtoring may be on a yearly basis. ' ' ^ \. 

Modifjcatiw and updating may vary with the awareness of the need for 
change by the ins.tructional staff. 

For ongoing programs it would appear that Advisory Committees should 
meet at least once a year. 

Meetings & Reports: / 

An agenda should be developed and given to ^ committee member*:: in advance 
of .meeting. 

Accurate minutes should be- taken and disseminated as soon as possible to 
all members of the committee. ^ ' ' 0 ■ 

A file of the minutes and other pertinant data should be established 
and maintained. 

Recommendations: 

When action is indicated this should be transmitted td the appropriate 
administrator post-haste. , / ^• 

In general, administrative, personnel "hnd faculty should be kept abreast 
of all developments. 

■\ ' . ■ 

informing the public: ^ 

When applicable, students and the community served by the institution - 
should be made aware of what is going oa. 

Recognition: 

It is imperative that non-institutional members services should be 
acknowledged at appropriate times. 

References : , * ^, 

^ Handbook for Members of Consultant Committers for Occupations : " 
University of the State of New York, the State Education Department 
office of Occupational and Continuirtg Education, Albany, New York. 

:The Advisory Committee : a handbook on how to schedule and conduct meetings 
Los Angeles City Schools Division of Career and Continuing Education, 
Career Education Services Unit, Industrial Education Office, Los Angeles, 
California. 

Advi sory Commi ttees : office of the fJapa County Superintendents office, 
fci^pa, California. 

• Function of a Technical-Vocational Advisory Committee : South County 
Commumty College District, Chabot College, Hayward, California. 
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